Reward Shaping and Mixed Resolution Function Approximation

In contrast to supervised learning, RL agents are not given instructive feedback on what the best decision in a particular situation is. This leads to the temporal credit assignment problem, that is, the problem of determining which part of the behaviour deserves the reward (Sutton, 1984). To address this issue, the iterative approach to RL applies backpropagation of the value function in the state space. Because this is a delayed, iterative technique, it usually leads to a slow convergence, especially when the state space is huge. In fact, the state space grows exponentially with each variable added to the encoding of the environment when the Markov property needs to be preserved (Sutton & Barto, 1998). When the state space is huge, the tabular representation of the value function with a separate entry for each state or state-action pair becomes ABSTRACT